Local and Global Regressive Mapping for Manifold Learning with Out-of-Sample Extrapolation

Over the past few years, a large family of manifold learning algorithms have been proposed, and applied to various applications. While designing new manifold learning algorithms has attracted much research attention, fewer research efforts have been focused on out-ofsample extrapolation of learned manifold. In this paper, we propose a novel algorithm of manifold learning. The proposed algorithm, namely Local and Global Regressive Mapping (LGRM), employs local regression models to grasp the manifold structure. We additionally impose a global regression term as regularization to learn a model for out-of-sample data extrapolation. Based on the algorithm, we propose a new manifold learning framework. Our framework can be applied to any manifold learning algorithms to simultaneously learn the low dimensional embedding of the training data and a model which provides explicit mapping of the outof-sample data to the learned manifold. Experiments demonstrate that the proposed framework uncover the manifold structure precisely and can be freely applied to unseen data. Introduction & Related Works Unsupervised dimension reduction plays an important role in many applications. Among them, manifold learning, a family of non-linear dimension reduction algorithms, has attracted much attention. During recent decade, researchers have developed various manifold learning algorithms, such as ISOMap (Tenenbaum, Silva, & Langford 2000), Local Linear Embedding (LLE) (Roweis & Saul 2000), Laplacian Eigenmap (LE) (Belkin & Niyogi 2003), Local Tangent Space Alignment (LTSA) (Zhang & Zha 2004), Local Spline Embedding (LSE) (Xiang et al. 2009), etc . Manifold learning has been applied to different applications, particularly in the field of computer vision, where it has been experimentally demonstrated that linear dimension reduction methods are not capable to cope with the data sampled from non-linear manifold (Chin & Suter 2008). Suppose there are n training data X = {x1, ..., xn} densely sampled from smooth manifold, where xi ∈ R for 1 ≤ i ≤ n. Denote Y = {y1, ..., yn}, where yi ∈ R(m < d) Copyright c © 2010, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. is the low dimensional embedding of xi. We define Y = [y1, ..., yn] as the low dimensional embedding matrix. Although the motivation of manifold learning algorithm differs from one to another, the objective function of ISOMap, LLE and LE can be uniformly formulated as follows (Yan et al. 2005). min Y T BY =I tr(Y T LY ), (1) where tr(·) is the trace operator, B is a constraint matrix, and L is the Laplacian matrix computed according to different criterions. It is also easy to see that (1) generalizes the objective function of other manifold learning algorithms, such as LTSA. Clearly, the Laplacian matrix plays a key role in manifold learning. Different from linear dimension reduction approaches, most of the manifold learning algorithms do not provide explicit mapping of the unseen data. As a compromise, Locality Preserving Projection (LPP) (He & Niyogi 2003) and Spectral Regression (SR) (Cai, He, & Han 2007) were proposed, which introduce linear projection matrix to LE. However, because a linear constraint is imposed, both algorithms fail in preserving the intrinsical non-linear structure of the data manifold. Manifold learning algorithms can be described as Kernel Principal Component Analysis (KPCA) (Schölkopf, Smola, & Müller 1998) on specially constructed Gram matrices (Ham et al. 2004). According to the specific algorithmic procedures of manifold learning algorithms, Bengio et al. have defined a data dependent kernel matrix K for ISOMap, LLE and LE, respectively (Bengio et al. 2003). Given the data dependent kernel matrix K, out-of-sample data can be extrapolated by employing Nyström formula. The framework proposed in (Bengio et al. 2003) generalizes Landmark ISOMap (Silva & Tenenbaum 2003). Similar algorithm was also proposed in (Chin & Suter 2008) for Maximum Variance Unfolding (MVU). Note that Semi-Definite Programming is conducted in MVU. It is very time consuming and thus less practical. One limitation of this family of algorithms is that the design of data dependent kernel matrices for various manifold learning algorithms is a nontrivial task. For example, compared with LE, it is not that straightforward to define the data dependent kernel matrix for LLE (Bengio et al. 2003) and it still remains unclear how to define the kernel matrices for other manifold learning algorithms, i.e., LTSA. In (Saul & Roweis 2003), a nonparametric approach was proposed for out-of-sample extrapolation of LLE. Let xo ∈ R be the novel data to be extrapolated and Xo = {xo1, ..., xok} ⊂ X be a set of data which are k nearest neighbor set of xo in R. The low dimensional embedding yo of xo is given by ∑k i=1 wiyoi, in which yoi is the low dimensional embedding of xoi and wi (1 ≤ i ≤ k) can be obtained by minimizing the following objective function.